Method and system for inline deduplication

ABSTRACT

A method for storing data includes receiving, by a data cluster, a request to store data from a host, deduplicating, by the data cluster, the data to obtain deduplicated data on a first data node, replicating the deduplicated data to generate a plurality of replicas, and storing a first replica of the plurality of replicas on a second data node and a second replica of the plurality of replicas on a third data node, wherein the first data node, the second data node and the third data node are in the data cluster.

BACKGROUND

Computing devices may include any number of internal components such asprocessors, memory, and persistent storage. Each of the internalcomponents of a computing device may be used to generate data. Theprocess of generating, storing, and backing-up data may utilizecomputing resources of the computing devices such as processing andstorage. The utilization of the aforementioned computing resources togenerate backups may impact the overall performance of the computingresources.

SUMMARY

In general, in one aspect, the invention relates to a method for storingdata. The method includes receiving, by a data cluster, a request tostore data from a host, deduplicating, by the data cluster, the data toobtain deduplicated data on a first data node, replicating thededuplicated data to generate a plurality of replicas, and storing afirst replica of the plurality of replicas on a second data node and asecond replica of the plurality of replicas on a third data node,wherein the first data node, the second data node and the third datanode are in the data cluster.

In one aspect, non-transitory computer readable medium comprisingcomputer readable program code, which when executed by a computerprocessor enables the computer processor to perform a method forgenerating data. The method includes receiving, by a data cluster, arequest to store data from a host, deduplicating, by the data cluster,the data to obtain deduplicated data on a first data node, replicatingthe deduplicated data to generate a plurality of replicas, and storing afirst replica of the plurality of replicas on a second data node and asecond replica of the plurality of replicas on a third data node,wherein the first data node, the second data node and the third datanode are in the data cluster.

In one aspect, a data cluster in accordance with one or more embodimentsof the invention includes a plurality of data nodes comprising a firstdata node, a second data node, and a third data node, wherein the firstdata node of the plurality node is programmed to receive a request tostore data from a host, deduplicate the data to obtain deduplicateddata, replicate the deduplicated data to generate a plurality ofreplicas, and initiate the storage of a first replica of the pluralityof replicas on the second data node of the plurality of nodes and asecond replica of the plurality of replicas on the third data node ofthe plurality of nodes.

BRIEF DESCRIPTION OF DRAWINGS

Certain embodiments of the invention will be described with reference tothe accompanying drawings. However, the accompanying drawings illustrateonly certain aspects or implementations of the invention by way ofexample and are not meant to limit the scope of the claims.

FIG. 1A shows a diagram of a system in accordance with one or moreembodiments of the invention.

FIG. 1B shows a diagram of a first data cluster in accordance with oneor more embodiments of the invention.

FIG. 1C shows a diagram of another data cluster in accordance with oneor more embodiments of the invention.

FIG. 2A shows a flowchart for storing data in a data cluster inaccordance with one or more embodiments of the invention.

FIGS. 2B-2D show a first example in accordance with one or moreembodiments of the invention.

FIG. 3A shows a flowchart for storing data in a data cluster inaccordance with one or more embodiments of the invention.

FIGS. 3B-3D show a second example in accordance with one or moreembodiments of the invention.

FIG. 4 shows a diagram of a computing device in accordance with one ormore embodiments of the invention.

DETAILED DESCRIPTION

Specific embodiments will now be described with reference to theaccompanying figures. In the following description, numerous details areset forth as examples of the invention. It will be understood by thoseskilled in the art that one or more embodiments of the present inventionmay be practiced without these specific details and that numerousvariations or modifications may be possible without departing from thescope of the invention. Certain details known to those of ordinary skillin the art are omitted to avoid obscuring the description.

In the following description of the figures, any component describedwith regard to a figure, in various embodiments of the invention, may beequivalent to one or more like-named components described with regard toany other figure. For brevity, descriptions of these components will notbe repeated with regard to each figure. Thus, each and every embodimentof the components of each figure is incorporated by reference andassumed to be optionally present within every other figure having one ormore like-named components. Additionally, in accordance with variousembodiments of the invention, any description of the components of afigure is to be interpreted as an optional embodiment, which may beimplemented in addition to, in conjunction with, or in place of theembodiments described with regard to a corresponding like-namedcomponent in any other figure.

Throughout this application, elements of figures may be labeled as A toN. As used herein, the aforementioned labeling means that the elementmay include any number of items and does not require that the elementinclude the same number of elements as any other item labeled as A to N.For example, a data structure may include a first element labeled as Aand a second element labeled as N. This labeling convention means thatthe data structure may include any number of the elements. A second datastructure, also labeled as A to N, may also include any number ofelements. The number of elements of the first data structure and thenumber of elements of the second data structure may be the same ordifferent.

In general, embodiments of the invention relate to a method and systemfor storing data in a data cluster. Embodiments of the invention mayutilize a deduplicator that performs deduplication on data obtained froma host to generate deduplicated data. In one or more embodiments of theinvention, the deduplicated data is subsequently replicated to othernodes in the data cluster. Embodiments of the invention may includesending a confirmation to the host that the obtained data isdeduplicated and stored in the data cluster.

FIG. 1A shows an example system in accordance with one or moreembodiments of the invention. The system includes a host (100) and adata cluster (110). The host (100) is operably connected to the datacluster (110) via any combination of wired and/or wireless connections.

In one or more embodiments of the invention, the host (100) utilizes thedata cluster (110) to store data. The data stored may be, for example,backups of databases, files, applications, and/or other types of datawithout departing from the invention.

In one or more embodiments of the invention, the host (100) isimplemented as a computing device (see e.g., FIG. 4). The computingdevice may be, for example, a laptop computer, a desktop computer, aserver, a distributed computing system, or a cloud resource (e.g., athird-party storage system accessible via a wired or wirelessconnection). The computing device may include one or more processors,memory (e.g., random access memory), and persistent storage (e.g., diskdrives, solid state drives, etc.). The computing device may includeinstructions, stored on the persistent storage, that when executed bythe processor(s) of the computing device cause the computing device toperform the functionality of the host (100) described throughout thisapplication.

In one or more embodiments of the invention, the host (100) isimplemented as a logical device. The logical device may utilize thecomputing resources of any number of computing devices and therebyprovide the functionality of the host (100) described throughout thisapplication.

In one or more embodiments of the invention, the data cluster (110)stores data and/or backups of data generated by the host (100). The dataand/or backups of data may be deduplicated versions of data obtainedfrom the host. The data cluster may replicate the deduplicated versionsof the data (also referred to as deduplicated data) to nodes operatingin the data cluster (110).

As used herein, deduplication refers to methods of storing only portionsof files (also referred to as file segments or segments) that are notalready stored in persistent storage. For example, when multipleversions of a large file, having only minimal differences between eachof the versions, are stored without deduplication, storing each versionwill require approximately the same amount of storage space of apersistent storage. In contrast, when the multiple versions of the largefile are stored with deduplication, only the first version of themultiple versions stored will require a substantial amount of storage.Once the first version is stored in the persistent storage, thesubsequent versions of the large file subsequently stored will bede-duplicated before being stored in the persistent storage resulting inmuch less storage space of the persistent storage being required tostore the subsequently stored versions when compared to the amount ofstorage space of the persistent storage required to store the firststored version.

Continuing with the discussion of FIG. 1A, the data cluster (110) mayinclude nodes that each store any number of deduplicated data. The datamay be obtained by other nodes (through replications) or obtained fromthe host (100). For additional details regarding the data cluster (110),see, e.g., FIGS. 1B and 1C.

FIG. 1B shows a diagram of a data cluster (120) in accordance with oneor more embodiments of the invention. The data cluster (120) may be anembodiment of the data cluster (110, FIG. 1A) discussed above. The datacluster (120) may include a deduplicator (122) and any number of datanodes (124, 126). The components of the data cluster (120) may beoperably connected via any combination of wired and/or wirelessconnections. Each of the aforementioned components is discussed below.

In one or more embodiments of the invention, the deduplicator(s) (122)is a device (physical or logical) that includes functionality to performdeduplication on data obtained from a host (e.g., 100, FIG. 1A). Thededuplicator (122) may store information useful to perform theaforementioned functionality. The information may include deduplicationidentifiers (D-IDs). A D-ID is a unique identifier that identifiesportions of data (e.g., segments) that are stored in the data cluster(120). The D-ID may be used to determine whether a data segment ofobtained data is already present elsewhere in the data cluster (120).The deduplicator (122) may use the information to perform thededuplication on the obtained data to generate deduplicated data. Afterdeduplication, the deduplicated data may be replicated to the data nodes(124, 126) a predetermined number of times. The deduplicator (122) mayperform the deduplication/replication via the method illustrated in FIG.2A.

In one or more of embodiments of the invention, the deduplicator (122)is implemented as computer instructions, e.g., computer code, stored ona persistent storage that when executed by a processor of a data node(e.g., 124, 126) cause the data node to provide the aforementionedfunctionality of the deduplicator (122) described throughout thisapplication and/or all, or a portion thereof, of the method illustratedin FIG. 2A.

In one or more embodiments of the invention, the deduplicator (122) isimplemented as a computing device (see e.g., FIG. 4). The computingdevice may be, for example, a laptop computer, a desktop computer, aserver, a distributed computing system, or a cloud resource (e.g., athird-party storage system accessible via a wired or wirelessconnection). The computing device may include one or more processors,memory (e.g., random access memory), and persistent storage (e.g., diskdrives, solid state drives, etc.). The computing device may includeinstructions, stored on the persistent storage, that when executed bythe processor(s) of the computing device cause the computing device toperform the functionality of the deduplicator (122) described throughoutthis application and/or all, or a portion thereof, of the methodillustrated in FIG. 2A.

In one or more embodiments of the invention, the deduplicator (122) isimplemented as a logical device. The logical device may utilize thecomputing resources of any number of computing devices and therebyprovide the functionality of the deduplicator (122) described throughoutthis application and/or all, or a portion thereof, of the methodillustrated in FIG. 2A.

In one or more embodiments of the invention, the data nodes (124, 126)are devices that store deduplicated data. The data nodes (124, 126) mayinclude persistent storage that may be used to store the deduplicateddata.

In one or more embodiments of the invention, each data node (124, 126)is implemented as a computing device (see e.g., FIG. 4). The computingdevice may be, for example, a laptop computer, a desktop computer, aserver, a distributed computing system, or a cloud resource (e.g., athird-party storage system accessible via a wired or wirelessconnection). The computing device may include one or more processors,memory (e.g., random access memory), and persistent storage (e.g., diskdrives, solid state drives, etc.). The computing device may includeinstructions, stored on the persistent storage, that when executed bythe processor(s) of the computing device cause the computing device toperform the functionality of the data node (124, 126) describedthroughout this application and/or all, or a portion thereof, of themethod illustrated in FIG. 2A.

In one or more embodiments of the invention, the data nodes (124, 126)are implemented as a logical device. The logical device may utilize thecomputing resources of any number of computing devices and therebyprovide the functionality of the data nodes (124, 126) describedthroughout this application and/or all, or a portion thereof, of themethod illustrated in FIG. 2B.

FIG. 1C shows a diagram of another example data cluster (130) inaccordance with one or more embodiments of the invention. The datacluster (130) may be an embodiment of the data cluster (110, FIG. 1A)discussed above. The data cluster (130) may include an accelerator pool(140) and a non-accelerator pool (150). The accelerator pool (140) mayinclude a deduplicator(s) (142) and any number of data nodes (144, 146).Similarly, the non-accelerator pool (150) includes any number of datanodes (154, 156). The components of the data cluster (130) may beoperably connected via any combination of wired and/or wirelessconnections. Each of the aforementioned components is discussed below.

In one or more embodiments of the invention, the deduplicator (142) is adevice that includes functionality to perform deduplication on dataobtained from a host (e.g., 100, FIG. 1A). The deduplicator (142) maystore information useful to perform the aforementioned functionality.The information may include deduplication identifiers (D-IDs). A D-ID isa unique identifier that identifies portions of the data (e.g.,segments) that are stored in the data cluster (130). The D-ID may beused to determine whether a data segment of obtained data is alreadypresent elsewhere in the accelerator pool (140) or the non-acceleratorpool (150). The deduplicator (142) may use the information to performthe deduplication and generate deduplicated data. After deduplication,the deduplicated data may be replicated to the non-accelerated pool(150) a predetermined number of times. The deduplicator (142) mayperform the deduplication/replication via the method illustrated in FIG.3A.

In one or more of embodiments of the invention, the deduplicator (142)is implemented as computer instructions, e.g., computer code, stored ona persistent storage that when executed by a processor of a data node(e.g., 144, 146) of the accelerator pool (140) cause the data node toprovide the aforementioned functionality of the deduplicator (142)described throughout this application and/or all, or a portion thereof,of the method illustrated in FIG. 3A.

In one or more embodiments of the invention, the deduplicator (142) isimplemented as a computing device (see e.g., FIG. 4). The computingdevice may be, for example, a laptop computer, a desktop computer, aserver, a distributed computing system, or a cloud resource (e.g., athird-party storage system accessible via a wired or wirelessconnection). The computing device may include one or more processors,memory (e.g., random access memory), and persistent storage (e.g., diskdrives, solid state drives, etc.). The computing device may includeinstructions, stored on the persistent storage, that when executed bythe processor(s) of the computing device cause the computing device toperform the functionality of the deduplicator (142) described throughoutthis application and/or all, or a portion thereof, of the methodillustrated in FIG. 3A.

In one or more embodiments of the invention, the deduplicator (142) isimplemented as a logical device. The logical device may utilize thecomputing resources of any number of computing devices and therebyprovide the functionality of the deduplicator (142) described throughoutthis application and/or all, or a portion thereof, of the methodillustrated in FIG. 3A.

Continuing with the discussion of FIG. 1C, different data nodes in thecluster may include different quantities and/or types of computingresources, e.g., processors providing processing resources, memoryproviding memory resources, storages providing storage resources,communicators providing communications resources. Thus, the system mayinclude a heterogeneous population of nodes.

The heterogeneous population of nodes may be logically divided into anaccelerator pool (140) including nodes that have more computingresources, e.g., high performance nodes (144, 146) than other nodes anda non-accelerator pool (150) including nodes that have fewer computingresources, e.g., low performance nodes (154, 156) than the nodes in theaccelerator pool (140). For example, nodes of the accelerator pool (140)may include enterprise class solid state storage resources that providevery high storage bandwidth, low latency, and high input-outputs persecond (IOPS). In contrast, the nodes of the non-accelerator pool (150)may include hard disk drives that provide lower storage performance.While illustrated in FIG. 1C as being divided into two groups, the nodesmay be divided into any number of groupings based on the relativeperformance level of each node without departing from the invention.

In one or more embodiments of the invention, the data nodes (144, 146,154, 156) are devices that store deduplicated data. The data nodes (144,146, 154, 156) may include persistent storage that may be used to storethe deduplicated data.

In one or more embodiments of the invention, the non-accelerator pool(150) includes any number of fault domains. In one or more embodimentsof the invention, a fault domain is a logical grouping of nodes (e.g.,data nodes) that, when one node of the logical grouping of nodes goesoffline and/or otherwise becomes inaccessible, the other nodes in thelogical grouping of nodes are directly affected. The effect of the nodegoing offline to the other nodes may include the other nodes also goingoffline and/or otherwise inaccessible. The non-accelerator pool (150)may include multiple fault domains. In this manner, the events of onefault domain in the non-accelerator pool (150) may have no effect toother fault domains in the non-accelerator pool (150).

For example, two data nodes may be in a first fault domain. If one ofthese data nodes in the first fault domain experiences an unexpectedshutdown, other nodes in the first fault domain may be affected. Incontrast, another data node in the second fault domain may not beaffected by the unexpected shutdown of a data node in the first faultdomain. In one or more embodiments of the invention, the unexpectedshutdown of one fault domain does not affect the nodes of other faultdomains. In this manner, data may be replicated and stored acrossmultiple fault domains to allow high availability of the data.

In one or more embodiments of the invention, each data node (144, 146,154, 156) is implemented as a computing device (see e.g., FIG. 4). Thecomputing device may be, for example, a laptop computer, a desktopcomputer, a server, a distributed computing system, or a cloud resource(e.g., a third-party storage system accessible via a wired or wirelessconnection). The computing device may include one or more processors,memory (e.g., random access memory), and persistent storage (e.g., diskdrives, solid state drives, etc.). The computing device may includeinstructions, stored on the persistent storage, that when executed bythe processor(s) of the computing device cause the computing device toperform the functionality of the data node (144, 146, 154, 156)described throughout this application and/or all, or a portion thereof,of the method illustrated in FIG. 3A.

In one or more embodiments of the invention, the data nodes (144, 146,154, 156) are implemented as a logical device. The logical device mayutilize the computing resources of any number of computing devices andthereby provide the functionality of the data nodes (144, 146, 154, 156)described throughout this application and/or all, or a portion thereof,of the method illustrated in FIG. 3A.

FIG. 2A shows a flowchart for storing data in a data cluster inaccordance with one or more embodiments of the invention. The methodshown in FIG. 2A may be performed by, for example, a deduplicator (122,FIG. 1B). Other components of the system illustrated in FIGS. 1A and 1Bmay perform the method of FIG. 2A without departing from the invention.While the various steps in the flowchart are presented and describedsequentially, one of ordinary skill in the relevant art will appreciatethat some or all of the steps may be executed in different orders, maybe combined or omitted, and some or all steps may be executed inparallel.

Turning to FIG. 2A, in step 220, data is obtained from a host. The datamay be a file, a file segment, a collection of files, or any other typeof data without departing from the invention. The data may include oneor more data segments. The data may be obtained in response to a requestto store data and/or backup the data. Other requests may be used toinitiate the method without departing from the invention.

In step 222, deduplication is performed on the obtained data to obtaindeduplicated data. In one or more embodiments of the invention, thededuplication is performed by identifying data segments of data in theobtained data and assigning a fingerprint to each data segment. Afingerprint is a unique identifier that may be stored in metadata of theobtained data. The deduplicator, when performing the deduplication, maygenerate a fingerprint for a data segment of the obtained data andidentify whether the fingerprint matches an existing fingerprint storedin the deduplicator. If the fingerprint matches an existing fingerprint,the data segment associated with the data segment may be deleted, as itis already stored in the data cluster. If the fingerprint does not matchany existing fingerprints, the data segment may be stored as part of thededuplicated data. Additionally, the fingerprint may be stored in thededuplicator for future use.

In one or more embodiments of the invention, the process of generating afingerprint for a data segment of the obtained data may be repeated forall data segments in the obtained data. The process may result in thegeneration of deduplicated data.

In step 224, a number (N) of replicas of deduplicated data to generateis determined. In one or more embodiments of the invention, the number(N) is obtained from the host. The host may request that N replicas ofthe deduplicated data be stored in the data cluster. In such scenarios,step 224 may be performed whenever the number N is obtained from thehost.

In one or more embodiments of the invention, the deduplicator determinesthe number (N) by querying the host to obtain the number (N). The number(N) may be based on a request by a user to replicate the data in thedata cluster a predetermined amount of times. The user may operate aclient (i.e., a computing device used by the user and operativelyconnected to the host) to send the request for the number of replicas tothe host.

In another embodiment of the invention, the deduplicator includesinformation about a default number of replicas to generate.

In step 226, N−1 replicas of the deduplicated data are generated in thedata cluster. In one or more embodiments of the invention, thededuplicated data generated in step 222 is the first deduplicated dataof the N number of deduplicated data. The deduplicated data may bereplicated N−1 more times. This results in N total deduplicated datastored in the data cluster.

In one or more embodiments of the invention, each deduplicated datagenerated is stored in a data node of the data cluster. The data node ofdeduplicated data is generated by copying the first (or anotherpreviously generated) deduplicated data. In this manner, the obtaineddata only has to be deduplicated once and then the resultingdeduplicated data may itself be copied (i.e., replicated) to generatethe remaining N−1 replicas that have been requested.

In step 228, confirmation is sent to the host. In one or moreembodiments of the invention, the confirmation is an acknowledgement(ACK) that confirms either: (i) receipt of the data by the data cluster,and/or (ii) completion of the deduplication and requested replication ofthe data stored in the data cluster.

Example 1

The following section describes an example. The example is not intendedto limit the invention. The example is illustrated in FIGS. 2B-2D, withFIG. 2B showing an example system at a first point in time. Turning tothe example, consider a scenario in which a host (200) wants to storededuplicated versions of host data (202) in a data cluster (210). Thedata cluster (210), which includes a deduplicator (212), obtains thehost data (202) in a data node A (214) (see e.g., FIG. 2A, Step 220).

The deduplicator (212) may further perform the method of FIG. 2A todeduplicate the obtained host data (202) (see e.g., FIG. 2A, Step 222).The process of deduplicating the host data (202) may include identifyingdata segments in the host data (202) and generating fingerprints of eachdata segment. For each generated fingerprint, the deduplicator (212) maysearch data stored in the deduplicator (212) to determine whether thegenerated fingerprint matches an existing fingerprint in thededuplicator. For every generated fingerprint matching an existingfingerprint, the data segment associated with the generated fingerprintmay be deleted from the deduplicator. In this manner, the result isdeduplicated data with data segments not already stored in the datanodes (214, 216, 218) of the data cluster (210).

FIG. 2C shows the example system at a second point in time. At thesecond point in time, the data cluster (210) includes deduplicated data(204) stored in data node A (214). The deduplicator (212), continuingthe method of FIG. 2A, may determine a number (N) of deduplicated datato store in the data cluster (210) (see e.g., FIG. 2A, Step 224). Thenumber (N) may be determined based on a request from the host. The hostmay request that the number (N) be three. In other words, the hostrequests that three deduplicated data be stored in the data cluster(210).

Because one of the deduplicated data (204) is already generated in thispoint in time, the deduplicator may replicate the deduplicated data twomore times. Data node A (214), which stores the first deduplicated data(204), may replicate the deduplicated data (204) to data nodes B and C(216, 218) in parallel. In other words, the data node (214) may generatea second deduplicated data and a third deduplicated data at the sametime. The replicated data may be generated and stored serially withoutdeparting from the invention. FIG. 2D shows the example system at athird point in time. At this third point in time, deduplicated data B(206) and C (208) may be stored in data nodes B (216) and C (218),respectively (see e.g., FIG. 2A, Step 226). The deduplicator maycomplete performing the method of FIG. 2A by sending the confirmation ofstorage to the host.

End of Example 1

FIG. 3A shows a flowchart for storing data in a data cluster inaccordance with one or more embodiments of the invention. The methodshown in FIG. 3A may be performed by, for example, a deduplicator (142,FIG. 1C). Other components of the system illustrated in FIGS. 1A and 1Cmay perform the method of FIG. 3A without departing from the invention.While the various steps in the flowchart are presented and describedsequentially, one of ordinary skill in the relevant art will appreciatethat some or all of the steps may be executed in different orders, maybe combined or omitted, and some or all steps may be executed inparallel.

In step 340, data is obtained from a host by a node in the acceleratorpool. The data may be a file, a file segment, a collection of files, orany other type of data without departing from the invention. The datamay include one or more data segments.

In step 342, confirmation is sent to the host. In one or moreembodiments of the invention, the confirmation is an acknowledgement(ACK) that confirms that the data processing has been completed by thedata cluster. At this stage, from the perspective of the host, the datahas been backed up. This is the case even though data cluster is stillperforming the method shown in FIG. 3A.

In step 344, deduplication is performed on the obtained data to obtaindeduplicated data. In one or more embodiments of the invention, thededuplication is performed by identifying data segments of the obtaineddata and assigning a fingerprint to each data segment. A fingerprint isa unique identifier that may be stored in metadata of the data. Thededuplicator, when performing the deduplication, may generate afingerprint for a data segment of the obtained data and identify whetherthe fingerprint matches an existing fingerprint stored in thededuplicator. If the fingerprint matches an existing fingerprint, thedata segment associated with the data segment may be deleted, as it isalready stored in the data cluster. If the fingerprint does not matchany existing fingerprints, the data segment may be stored as part of thededuplicated data. Additionally, the fingerprint may be stored in thededuplicator for future use.

In one or more embodiments of the invention, the process of generating afingerprint for a data segment of the obtained data may be repeated forall data segments in the obtained data. The process may result in thegeneration of deduplicated data.

In step 346, a number (N) of replicas of deduplicated data to generateis determined. In one or more embodiments of the invention, the number(N) is obtained from the host. The host may request that N replicas ofthe deduplicated data be stored in the data cluster. In such scenarios,step 346 may be performed whenever the number N is obtained from thehost.

In one or more embodiments of the invention, the deduplicator determinesthe number (N) by querying the host to obtain the number (N). The number(N) may be based on a request by a user to replicate the data in thedata cluster a predetermined amount of times. The user may operate aclient (i.e., a computing device used by the user and operativelyconnected to the host) to send the request for the number of replicas tothe host.

In another embodiment of the invention, the deduplicator includesinformation about a default number of replicas to generate.

In step 348, N−1 replicas of the deduplicated data are generated andstored in the non-accelerator pool. In one or more embodiments of theinvention, the deduplicated data generated in step 344 is the firstdeduplicated data of the N number of deduplicated data. The deduplicateddata may be replicated N−1 more times. This results in N totaldeduplicated data stored in the data cluster.

In one or more embodiments of the invention, each deduplicated datagenerated is stored in a data node in the non-accelerator pool. In oneimplementation, each of the aforementioned data nodes in thenon-accelerator pool is in its own fault domain. In this manner, thededuplicated data may be stored across multiple fault domains in thenon-accelerated pool.

Example 2

The following section describes an example. The example is not intendedto limit the invention. The example is illustrated in FIGS. 3B-3D, withFIG. 3B showing an example system at a first point in time. Turning tothe example, consider a scenario in which a host (300) wants to storededuplicated versions of host data (302) in a data cluster (310). Thehost (300) may send the host data (302) to a data node (322) operatingon an accelerator pool (320).

The accelerator pool (320) may include a deduplicator (324) that may beutilized to perform the method of FIG. 3A to perform a deduplication onthe host data (302). The process of deduplicating the host data (302)may include identifying data segments in the host data (302) andgenerating fingerprints of each data segment. For each generatedfingerprint, the deduplicator (324) may search data stored in thededuplicator (324) to determine whether the generated fingerprintmatches an existing fingerprint in the deduplicator (324). For everygenerated fingerprint matching an existing fingerprint, the data segmentassociated with the generated fingerprint may be deleted. In thismanner, the result is deduplicated data with data segments not alreadystored in the data nodes (322, 332, 334) of the data cluster (310).

FIG. 3C shows the example system at a second point in time. In FIG. 3C,the deduplicated data (304) may be stored in the accelerator pool. Thededuplicator (324) may continue the method of FIG. 3A to replicate thededuplicated data (304) to data nodes (332, 334) in the non-acceleratorpool (330). The deduplicator (322) may determine a number of replicas togenerate by obtaining the number from the host (300). The host mayrespond with a request that the deduplicated data (304) be replicatedthree times in the data cluster (310).

Because one of the three requested deduplicated data is alreadygenerated in this point in time, the deduplicator may replicate thededuplicated data (304) two more times. Data node A (324), which storesthe first deduplicated data (304), may replicate the deduplicated data(304) to data nodes B and C (332, 334) in parallel. In other words, thedata node (314) may generate and store a second deduplicated data and athird deduplicated data at the same time. Alternatively, the replicasmay be generated and stored serially.

FIG. 3D shows the example system at a third point in time. At this thirdpoint in time, deduplicated data B (306) and C (308) may be stored indata nodes B (332) and C (334), respectively. Data node B (332) may be anode of a fault domain that is different from a fault domain in whichdata node C (334) is a part of. In this manner, if one of the two datanodes (332, 334) were to go through an unexpected shutdown, the otherdata node (332, 334) would not be affected, and the deduplicated data(e.g., 304, 306, 308) could still be accessed in the event of a recoveryrequest by the host (300).

End of Example 2

As discussed above, embodiments of the invention may be implementedusing computing devices. FIG. 4 shows a diagram of a computing device inaccordance with one or more embodiments of the invention. The computingdevice (400) may include one or more computer processors (402),non-persistent storage (404) (e.g., volatile memory, such as randomaccess memory (RAM), cache memory), persistent storage (406) (e.g., ahard disk, an optical drive such as a compact disk (CD) drive or digitalversatile disk (DVD) drive, a flash memory, etc.), a communicationinterface (412) (e.g., Bluetooth interface, infrared interface, networkinterface, optical interface, etc.), input devices (410), output devices(408), and numerous other elements (not shown) and functionalities. Eachof these components is described below.

In one embodiment of the invention, the computer processor(s) (402) maybe an integrated circuit for processing instructions. For example, thecomputer processor(s) may be one or more cores or micro-cores of aprocessor. The computing device (400) may also include one or more inputdevices (410), such as a touchscreen, keyboard, mouse, microphone,touchpad, electronic pen, or any other type of input device. Further,the communication interface (412) may include an integrated circuit forconnecting the computing device (400) to a network (not shown) (e.g., alocal area network (LAN), a wide area network (WAN) such as theInternet, mobile network, or any other type of network) and/or toanother device, such as another computing device.

In one embodiment of the invention, the computing device (400) mayinclude one or more output devices (408), such as a screen (e.g., aliquid crystal display (LCD), a plasma display, touchscreen, cathode raytube (CRT) monitor, projector, or other display device), a printer,external storage, or any other output device. One or more of the outputdevices may be the same or different from the input device(s). The inputand output device(s) may be locally or remotely connected to thecomputer processor(s) (402), non-persistent storage (404), andpersistent storage (406). Many different types of computing devicesexist, and the aforementioned input and output device(s) may take otherforms.

One or more embodiments of the invention may be implemented usinginstructions executed by one or more processors of the data managementdevice. Further, such instructions may correspond to computer readableinstructions that are stored on one or more non-transitory computerreadable mediums.

One or more embodiments of the invention may improve the operation ofone or more computing devices. More specifically, embodiments of theinvention improve the efficiency of performing storage and/or backupoperations in a data cluster. The efficiency improvement may be achievedby performing deduplication on data prior to replicating the data tonodes in the data cluster. By deduplicating prior to replicating, thedata cluster may reduce the amount of computing resources needed toperform replication over other systems that replicate the data prior todeduplication.

Further, embodiments of the invention improve the deduplication byupgrading the nodes performing a deduplication, i.e., performing thededuplication in the accelerator pool. The use of the higher-performancenodes in the accelerator pool may reduce processing time compared tonon-high-performance nodes. This upgrade, along with performing thededuplication prior to replicating the data, may reduce the total amountof time required to store the data and reduces the total use ofcomputing resources for a storage operation.

Thus, embodiments of the invention may address the problem ofinefficient use of computing resources. This problem arises due to thetechnological nature of the environment in which storage operations areperformed.

The problems discussed above should be understood as being examples ofproblems solved by embodiments of the invention disclosed herein and theinvention should not be limited to solving the same/similar problems.The disclosed invention is broadly applicable to address a range ofproblems beyond those discussed herein.

While the invention has been described above with respect to a limitednumber of embodiments, those skilled in the art, having the benefit ofthis disclosure, will appreciate that other embodiments can be devisedwhich do not depart from the scope of the invention as disclosed herein.Accordingly, the scope of the invention should be limited only by theattached claims.

What is claimed is:
 1. A method for managing data, the methodcomprising: receiving, by a first data node in a data cluster, a requestto store data from a host; providing, by the first data node, the datato a deduplicator in the data cluster, wherein the deduplicator isoperatively connected to the first data node; deduplicating, by thededuplicator in the data cluster, the data to obtain deduplicated data;storing, by the deduplicator, the deduplicated data on the first datanode; after the deduplicated data is stored on the first data node:replicating, by the first data node, the deduplicated data to generate aplurality of replicas, wherein each of the plurality of replicascomprises a copy of the deduplicated data; initiating a storage of afirst replica of the plurality of replicas to a second data node and asecond replica of the plurality of replicas to a third data node,wherein the first data node, the second data node and the third datanode are in the data cluster, wherein the first replica is stored on thesecond data node and the second replica is stored on the third data nodein parallel, wherein the data cluster comprises an accelerator pool anda non-accelerator pool, and wherein the first data node and thededuplicator are in the accelerator pool and the second and third datanodes are in the non-accelerator pool.
 2. The method of claim 1, furthercomprising: sending, in response to the request, a confirmation to thehost that the request has been serviced.
 3. The method of claim 1,further comprising: determining a number (N) of replicas to generate,wherein N is a positive integer greater than 2, and wherein replicatingthe deduplicated data to generate the plurality of replicas comprisesgenerating N−1 replicas.
 4. The method of claim 1, wherein thededuplicator is executing on a fourth data node in the data cluster. 5.A data cluster, comprising: a deduplicator; a plurality of data nodescomprising a first data node, a second data node, and a third data node;wherein the deduplicator is programmed to: obtain data to deduplicatefor the first data node; deduplicate the data to obtain deduplicateddata; and providing the deduplicated data on the first data node,wherein the first data node of the plurality of data nodes is programmedto: receive a request to store the data from a host; provide the data tothe deduplicator, wherein the deduplicator is operatively connected tothe first data node; receive the deduplicated data from thededuplicator; store the deduplicated data; after the deduplicated datais stored on the first data node: replicate the deduplicated data togenerate a plurality of replicas, wherein each of the plurality ofreplicas comprises a copy of the deduplicated data; initiate storage ofa first replica of the plurality of replicas on the second data node ofthe plurality of data nodes and a second replica of the plurality ofreplicas on the third data node of the plurality of data nodes, whereinthe first replica is stored on the second data node and the secondreplica is stored on the third data node in parallel, wherein the datacluster further comprises an accelerator pool and a non-acceleratorpool, and wherein the first data node and the deduplicator are in theaccelerator pool and the second and third data nodes are in thenon-accelerator pool.
 6. The data cluster of claim 5, wherein the firstdata node is further programmed to: send, in response to the request, aconfirmation to the host that the request has been serviced, wherein theconfirmation is sent after the first replica is stored on the seconddata node and the second replica is stored on the third data node. 7.The data cluster of claim 5, wherein the first data node is furtherprogrammed to: determining a number (N) of replicas to generate, whereinN is a positive integer greater than 2, and replicating the deduplicateddata to generate the plurality of replicas comprises generating N−1replicas.